Method and system for composing electronic music and generating graphical information

ABSTRACT

A method and system are provided that allows a user to interactively compose music and create a video of an animated character that dances to the rhythm and beat of the composed music. To compose music, the user interactively selects and arranges a plurality of music samples for a plurality of instruments. Music samples are independent pieces of music that can be seamlessly looped to compose the musical work. A harmonizer program allows the user to hammonize the music samples that play simultaneously, thereby allowing the user to compose a music piece without knowledge or experience in musical theory or training. The user creates a video of an animated character that dances to the rhythm and beat of the music using an animation program. The animation program provides the user with a plurality of video clips from which the user can select and arrange to create the video. Once completed, the user can play, edit, and save the composed music and the video.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a 371 of PCT/IB99/01655 filed Sep. 3, 1999 and alsoclaims the benefit of U.S. Provisional Application No. 60/099,290, filedSep. 4, 1998, the contents of which are hereby incorporated byreference.

FIELD OF THE INVENTION

The present invention relates generally to interactive computertechnology and, more particularly, to a method and system using acomputer to electronically compose music and generate correspondinggraphical animation.

BACKGROUND OF THE INVENTION

The composition of music is usually restricted to those individuals withmusical talent or, at a minimum, an understanding of music. Without oneor the other, an individual will usually find it difficult to composemusic successfully when measured by a common yardstick of taste. Thus,the widespread use of computers, while providing a wide variety of toolsto the expert musician, has done little to open doors for the musicallyunsophisticated to experience the joy of creating music. For example,software programs exist that allow a musician to compose, record, playback, and print music in various ways. These programs use, for example,digital audio and Musical Instruments Digital Interface (MIDI) thatallow a musician to hear music the second it is created, access a nearinfinite variety of musical instruments, and make major changes to acomposition with little effort. The problem with these tools is that theuser must still bring an understanding of music and/or talent into theprocess to create something that will satisfy oneself or others.

One solution allows those with little musical experience andsophistication to interact with a previously composed musical work. Anexample is described in U.S. Pat. No. 5,801,694. This patent discloses amethod in which a musical expert reduces a pre-existing musical workinto its elements such as, for example, a solo track and accompanyingtracks to create a musical template. These tracks are then stored in adatabase. Additional solo and accompanying tracks may be composed by theexpert musician and also stored in the database. The musicallyunsophisticated user can then construct a new arrangement of theexisting musical work by selecting various tracks from the database andallocating the tracks to fixed positions in the template. The templateallows a user to rearrange, for example, pieces of the accompanyingtrack while preserving the underlying structure of the musical workcomposed by the expert musician.

This method, however, provides too much structure for an unsophisticateduser interested in composing a unique musical work. It restricts thecreative process to selecting pieces from an expert musician'spre-composed tracks and placing them in positions defined by the expertmusician's template.

U.S. Pat. 5,679,913 discloses an electronic apparatus for automaticcomposition and reproduction of musical works. The electronic apparatuscomprises means for reading data of musical events on data tracks andmake the temporal lengths of selected data tracks uniform and tosynchronize in real time the reading of data tracks having equal ordifferent length. This document is considered with composition andreproduction of music only.

WO 97/21210 discloses a method and apparatus for interactively creatingmusical compositions based on a database with musical sequences and atemplate comprising a plurality of sequence positions. Using amenu-driven, graphical interface utilizing icons and images, a userinteractively selects a plurality of the fixed musical sequences, andallocates the selected sequences among the various fixed sequencepositions specified by the template. The result of this interactiveselection procedure is a musical composition that can be played. Thisdocument is also considered with composition and reproduction of musiconly, despite a graphical user interface is employed in the creating andplaying of music.

U.S. Pat. 5,005,459 discloses a musical tone visualising apparatusincluding an image memory for storing plural images relating to playersor musical instruments of an orchestra, a display unit for displayingthe images on a display screen thereof and a display control circuit forcontrolling the display unit so that displayed images can be varied inresponse to an inputted musical tone signal or inputted performanceinformation outputted from an electronic instrument. Thereby, thedisplayed images can be automatically varied in response to theperformance of the electronic musical instrument.

In light of the foregoing, there is a need for a method and system thatallows a musically unsophisticated user to experience the joy ofcomposing music.

SUMMARY OF THE INVENTION

Accordingly, the present invention is directed to a method for using acomputer to compose a musical work and create a video of an animatedcharacter. The present invention includes the steps of interactivelyselecting a music sample from a plurality of music samples, wherein eachof the music samples is an independent pre-composed piece of music offixed duration. A user then, interactively places the selected musicsample in a location in the musical work, and interactively repeats thestep of selecting a music sample and placing the selected music samplein a location in the musical work, as desired by the user, to create themusical work.

In another aspect, the invention includes a method for using a computerto compose a musical work for a plurality of instruments including thesteps of interactively selecting one of a plurality of musicalinstruments, interactively selecting one of a plurality of musicsamples, wherein each of the music samples is an independentpre-composed piece of music. A user then sequentially arranges theselected music samples. The user then interactively selects another oneof the plurality of musical instruments and interactively repeats thesteps of selecting a plurality of music samples and sequentiallyarranging the selected music samples for another one of the plurality ofmusical instruments. Next, the user interactively selects one of themusic samples previously arranged and interactively harmonizes thearranged music samples of all instruments that correspond in time to theselected music samples. The user interactively repeats the step ofselecting one of the music samples previously arranged, and harmonizingthe arranged music samples of all instruments that correspond in time tothe selected music sample previously arranged to compose the musicalwork.

In another aspect, the invention includes a system for composing amusical work comprising a central processing unit, a video display, aninput device, a general memory, means for interactively selecting aplurality of the music samples, interactively arranging the selectedmusic samples in a serial manner for a plurality of instruments, andinteractively selecting the music samples to be harmonized, as desiredby the user, and means for playing the musical work. The general memorystores a plurality of music samples, wherein each of the music samplesis an independent pre-composed piece of music, a harmonizer program thatharmonizes the music samples selected and arranged by a user, and aprogram that facilitates composition of the music.

The accompanying drawings are included to provide a furtherunderstanding of the invention and are incorporated in and constitute apart of this specification, illustrate several embodiments of theinvention and together with the description serve to explain theprinciples of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate an embodiment of the inventionand, together with the description, serve to explain the objects,advantages, and principles of the invention.

FIG. 1 is a diagram of a system according to an embodiment consistentwith the present invention.

FIG. 2 is a schematic diagram illustrating components of a deviceaccording to an embodiment of the present invention.

FIG. 3 is a block diagram illustrating the components of the generalmemory of a system according to one embodiment of the present invention.

FIG. 4 is a flowchart illustrating the process of the program thatfacilitates composing music and creating a video according to oneembodiment of the present invention.

FIG. 5 shows an image of a musical playroom according to one embodimentof the present invention.

FIG. 6 shows an image of the music composition screen according to oneembodiment of the present invention.

FIG. 7 shows the video composition screen according to one embodiment ofthe present invention.

FIG. 8 shows an image of the band performing music composed by a useraccording to one embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Reference will now be made in detail to the preferred embodiments of theinvention, examples of which are illustrated in the accompanyingdrawings.

FIG. 1 is a diagram of a system according to an embodiment consistentwith the present invention. A system 100 includes a computer 110connected to a monitor 120, a keyboard 130, a mouse 140, and a printer150. In the preferred embodiment, computer 110 is a multimedia systemhaving audio-visual capabilities. The present invention, however, may beimplemented in any computer platform with sufficient processing andmemory capacity to perform the processes described below.

FIG. 2 is a schematic diagram illustrating in detail components ofcomputer 110. A processor 200 operates on an operating system 210, whichmay be any operating system including Windows, DOS, MacOS, SONYPlaystation OS, Nintendo OS, OS2, and Linux. Computer 110 contains RAMcomposed of a general memory 220 and a display memory 222. Drivers 230control various input and output devices connected to computer 110.

FIG. 3 is a block diagram illustrating the components of general memory220 according to one embodiment of the present invention, which includesa program 310, a harmonizer program 320, and an animation program 330.Program 310 is a main program overseeing the operation of system 100.Harmonizer program 320 harmonizes musical instruments for musicalcomposition. Animation program 330 facilitates movements of a graphicalanimation figure according to a musical piece. Programs 310, 320, and330 may be written in languages such as C, C++, Be, Linux, Basic,VisualBasic, or Macromedia Director.

FIG. 4 is a flowchart illustrating the overall process of system 100according to one embodiment of the present invention. The process ofsystem 100 may be implemented, for example, as a story to addentertainment value and enhance ease of use. Program 310 may firstintroduce the background and characters of the story to the user (step405). Program 310 displays, for example, a background image, and in amultimedia system, also provides an audio narration. Program 310 alsopresents several options the user may select from. One option includes amusical playroom icon. If the user selects musical playroom icon (step410), program 310 displays a musical playroom with musicians and musicalinstruments as shown in FIG. 6.

In an embodiment consistent with the present invention, the user selectsany of the displayed musical instruments to compose music. When the userselects one of the instruments, for example, by clicking mouse 140 onthat instrument (step 415), program 310 displays a musical compositionscreen as shown in FIG. 7 (step 420). At this point, the user is free tocompose a musical piece using a harmony selector 710, a sample selector720, and an instrument row 730.

Specifically, sample selector 720 allows the user to choose a sample ina certain style and apply it to the music by clicking and dropping it inthe desired place in the work (step 430). Music samples are independentpieces of music composed by a musician and stored in memory. In apreferred embodiment, each sample is one measure long. Each music bar iscomposed with a similar interface so that it can be seamlessly looped toeach of the other music bars. This allows any music bar to be playedsequentially with any other music bar without a listener being able todistinguish the interface or hearing an abrupt break in the music. Inother words, the seamlessly looped music bars sound pleasing to thelistener when one is played after another.

Instrument row 730 displays the musical instrument selected for that row(step 435). In an embodiment consistent with the present invention, theuser may select and change the instrument of instrument row 730. Eachinstrument row has, for example, sixteen spaces for the user to placeselected music samples, corresponding to the standard sixteen measurelength of western music. A music sample cell 740 shows graphically thesample the user has selected, if any, to be played by the instrument inthe corresponding instrument row 730 at that given point and time in thecomposition. In a preferred embodiment, the memory stores a version ofeach sample played by each instrument. In an altemative embodiment,however, the samples can be “played” by different instruments by using,for example, a synthesizer to generate the sound of the selectedinstruments.

Music sample cell 740 also indicates the harmony for a particular columnsince all samples in the same column have the same harmony as describedbelow in connection with harmony selector 710 (step 440). The user maydelete a particular sample during composition, and insert instead a barof silence. A play button 750 activates program 310 to mix the composedmusic and play the music (step 445). Accordingly, system 110 facilitatesmusic composition by mixing the content of the different cells togetherto a piece of music, i.e., “folding” the columns and setting the rows insequence, thereby assembling all different pieces to one single track,which can be saved, edited, and played. Thus, system 100, throughharmonizer program 320, ensures a harmonic piece of music.

Harmony selector 710 allows the user to choose a harmony for each columnof instruments, thereby “harmonizing” all instruments shown incorresponding instrument row 730 (step 425). For example, if the userchooses the bass instrument (or any other) and then selects harmonyselector 710, harmonizer program 320 aligns all instruments to theselected harmony. In doing so, harmonizer program 320 drops off“disharmonies,” which may be defined, for example, as harmonies outsidewestern tonality and pop harmonies, and aligns the instrumentsautomatically. Thus, harmony selector 710 allows the user to compose amusic piece without knowledge or experience in musical theory ortraining.

In an embodiment consistent with the present invention, system 110stores a version of each music sample in a plurality of harmonies.Harmony selector 710 allows the user to select from, for example, sixdifferent harmonies for each music sample. A user selects a harmony foreach column of instruments by selecting the desired column and selectinga harmony from harmony selector 710. Alternatively, a user can placemusic samples, as desired, and harmonizer program 310 will harmonize acolumn to the harmony of the last music sample placed in the column.Harmonizer program 310 does so by retrieving the version of the musicsamples in the column with the harmony corresponding to the harmony ofthe last music sample placed in the column.

In another embodiment, the harmonizer program 310 may be software thatanalyzes the harmony of the selected music samples. In this embodiment,system 110 stores only one version of each music sample. The harmonizerprogram 310 transposes the music samples in each column to match theharmony selected by the user by changing, for example, the pitch of themusic samples.

Program 310, harmonizer program 320, and animation program 330 allow auser to compose a new musical work following any procedure the userdesires. A user at the music composition screen shown in FIG. 7 could,for example, choose an instrument 730, select a plurality of musicsamples 720, and arrange them in the chosen instruments instrument row740, as desired. The user could also, for example, select a music sampleand place the selected music sample in any space of any instrument'sinstrument row. Selection and placement of music samples in this mannercan continue, as the user desires, until all spaces for all instrumentsare filled or until the user has determined that the musical work iscomplete. Harmonization of each of the columns of instruments may bedone at any time using harmonizer buttons 710. Program 310 allows theuser to save, edit, and play the music at any time during the process.

Program 310 also provides an option to the user to create a videocharacter that dances to the composed music (step 450). If the userclicks on video character 610 (FIG. 6), for example, program 310presents a choreography composition screen of FIG. 8 (step 455). Fromthis screen, the user may select a choreography movement selector 810containing a plurality of pre-drawn dance movement clips. The user mayselect the desired dance movement clip and drop it in a desired cell ina filmstrip 820 (step 460). Accordingly, when the user plays thecomposed music, animation program 330 produces an animation character610 that dances by sequentially displaying the dance movement clips offilmstrip 820 at corresponding speed and tempo of the music. In anembodiment consistent with the present invention, the animated dancercorresponds to the character of the lead singer.

System 100 enables the user to freely combine dance moves to a piece ofmusic. Although one skilled in the art may design animation program 330to any musical beat, in one embodiment consistent with the presentinvention, the video character matches a beat of 120 bpm (beats perminute). The choreography movement images are preferably pre-drawn,although the user may elect to modify or create customized images.Animation program 330 enables the movements of the video character toautomatically adjust to the music as one image interleaves smoothly withthe following image regardless of which movement clips are selected.Program 310 also enables the user to save, load, or edit filmstrip 820containing the selected movement clips.

Once a user has composed a musical work and combined dance moves for theanimated character, performance button 760 allows the user to combinethe musical work and the dancing animated character. In an embodimentconsistent with the present invention, the music is “played” by a band.As shown in FIG. 5, each instrument is played by a character in the band510 and the lead singer 520 is the animated character that dances to thebeat of the music.

It will be apparent to those skilled in the art that variousmodifications and variations can be made in the method and system of thepresent invention without departing from the spirit or scope of theinvention. Thus, it is intended that the present invention cover themodifications and variations of this invention provided they come withinthe scope of the appended claims and their equivalents.

What is claimed is:
 1. In a computer, a method of composing a multimediawork comprising the steps of: in response to a user interactionselecting (430) a music sample from a plurality of music samples,wherein each of the music samples is an independent pre-composed pieceof music of fixed duration having a common interface that allows themusic samples to be seamlessly looped; and placing (740) the selectedmusic sample in a location in the multimedia work, wherein the step ofselecting (430) a music sample and placing (740) the selected musicsample in a location in the multimedia work is repeated in response to auser interaction to create the multimedia work characterized in that themethod further comprises the steps of: in response to further userinteraction selecting (460) a pre-drawn dance movement clip from aplurality of dance movement clips; and placing the selected pre-drawndance movement clip (810) in a location (820) in the multimedia work;wherein the step of selecting a pre-drawn dance movement clip andplacing the selected pre-drawn dance movement in a location (820) in themultimedia work is repeated in response to further user interaction tocreate the multimedia work as a user composed multimedia work with ananimated character (610) that dances to a user created composition ofthe musical samples by playing (750) the selected music samples andsequentially displaying (FIG. 6) the dance movement clips atcorresponding speed and tempo of the music.
 2. The method of claim 1,further including a step of removing a selected music sample to insert asample of silence.
 3. The method of claim 1, wherein a video isinteractively created from a plurality of video clips and said video issynchronized to a beat of the musical work.
 4. The method of claim 1,wherein the step of selecting a music sample comprises the steps of:selecting one of a plurality of musical instruments; selecting one of aplurality of music samples for the selected instrument, wherein each ofthe music samples is an independent pre-composed piece of music, andsequentially arranging the selected music sample; and selecting anotherone of the plurality of musical instruments, repeating the steps ofselecting a plurality of music samples and sequentially arranging theselected music samples for said another one of the plurality of musicalinstruments.
 5. The method of claim 4, further comprising the steps of:selecting one of the music samples previously arranged, harmonizing thearranged music samples of all instruments that correspond in Uime tosaid selected one of the music samples previously arranged; andrepeating the step of selecting one of the music samples previouslyarranged, and harmonizing the arranged music samples of all instrumentsthat correspond in time to said selected one of the music samplespreviously arranged to compose the musical work.
 6. The method of claim1, wherein the steps are performed in response to user interactionsusing a graphical user interface.
 7. The method of claim 1, furtherincluding the steps of: storing a version of the plurality of musicsamples for each of the specified instruments.
 8. The method of claim 1,wherein video of a dancer is choreographed to the musical work so thatmovements of the dancer match a beat of the musical work.
 9. The methodof claim 1, wherein the video of the dancer is choreographed to match120 beats per minute.
 10. The method of claim 1, wherein the interactivesteps are implemented using a story.
 11. The method of claim 1, whereinthe music samples are one measure long.
 12. A computer system comprisingmeans for executing the method according to any of the claims 1 through11.